Papers with backbone models
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated outstanding performance in many downstream tasks due to their emergent and in-context learning abilities. |
| Approach: | They propose a method that considers LLMs as annotators for type-aware data augmentation in GEC tasks. |
| Outcome: | The proposed method can generate consistent and typeaware data, which could improve the performance of large language models. |
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual aspect-based sentiment analysis depend on translation tools. |
| Approach: | They propose a cross-lingual aspect-based sentiment analysis framework that leverages a large language model to generate pseudo-labelled data in target language. |
| Outcome: | The proposed approach outperforms translation-based approaches in six languages and five backbone models. |
Copied to clipboard
| Challenge: | Existing methods that align natural language with SQL Language underestimate inherent structural characteristics of SQL and lead to structure errors. |
| Approach: | They propose a retrieval-argument framework that aligns natural language with SQL Language and trains one encoder-decoder-based model to fit all questions. |
| Outcome: | The proposed framework improves accuracy and robustness of text-to-SQL generation on five datasets. |
Copied to clipboard
| Challenge: | Recent trends in learning monolingual and multilingual sentence embeddings are based on contrastive learning (CL) among an anchor, one positive and multiple negative instances. |
| Approach: | They propose to leverage multiple positives to improve learning of multilingual sentence embeddings by using an anchor, one positive, and multiple negative instances. |
| Outcome: | The proposed approach improves retrieval, semantic similarity, and classification performance on unseen languages. |
Copied to clipboard
| Challenge: | Large pre-trained language models have been applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model. |
| Approach: | They propose to map programming language modalities into the same embedding space by separating embeddable spaces between modality and modality-relative training objectives. |
| Outcome: | The proposed model can be adapted and represented differently depending on which modality they belong to and to the ultimate benefit of the downstream task. |
Copied to clipboard
| Challenge: | Existing ESC datasets often use coarse-grained problem categories, limiting models’ ability to address users’ complex, overlapping challenges. |
| Approach: | They propose a generalizable fine-grained problem enhancement method that augments problem types, user scenarios, and profiles, enabling the construction of richer and more diverse ESC corpora. |
| Outcome: | The proposed method improves both automatic and human evaluation metrics across different models. |
Copied to clipboard
| Challenge: | Existing document-level relation extraction methods do not distinguish between mention-level features and entity-level feature . document-based methods are more challenging because of multiple mentions of entities. |
| Approach: | They propose a method which selectively attentions different entity mentions with respect to candidate relations and performs relation-specific representations of entities. |
| Outcome: | The proposed method improves relation-specific representations of entities on two benchmark datasets. |
Copied to clipboard
| Challenge: | Recent advances have underscored the potential of large language model (LLM)-based agents in financial decision-making. |
| Approach: | They propose to evaluate LLM agents using 13 different LLMs as backbone models across various market environments and tasks. |
| Outcome: | The proposed framework assesses the reasoning and decision-making capabilities of 13 different LLMs across various market environments and tasks. |
Copied to clipboard
| Challenge: | Existing approaches to the problem of open-retrieval conversational machine reading (OR-CMR) use two separate modules to approach the problem's two successive sub-tasks. |
| Approach: | They propose to model OR-CMR as a unified text-to-text task in a fully end-to end style and propose to use a text-based approach to solve the problem. |
| Outcome: | Experiments on the ShARC and OR-ShARC dataset show that the proposed framework can generalize to different backbone models. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) fail to effectively guide the planning trajectories during task solving and result in planning hallucinations. |
| Approach: | They propose a novel approach to enhance the planning capabilities of large language models by incorporating explicit action knowledge. |
| Outcome: | The proposed approach can achieve comparable or superior performance to existing baselines on HotpotQA and ALFWorld. |
Copied to clipboard
| Challenge: | Existing methods for automating medical coding focus on a single coding system . however, there are still challenges to overcome in coding. |
| Approach: | They propose a joint learning framework for Across Medical coding systems which jointly learns different coding system through multi-task learning. |
| Outcome: | The proposed framework improves the performance of the MIMIC-IV ICD-9 and MIMICIV I CD-10 datasets. |
Copied to clipboard
| Challenge: | Experiments on five text classification benchmarks and five backbone models have shown that our methods reduce the error rate over Mixup variants in a significant margin (up to 31.3%), especially in low-resource conditions (upto 17.5%). |
| Approach: | They propose to add a small adversarial perturbation to the mixing coefficients rather than the examples to relax locally linear constraints. |
| Outcome: | Experiments on five text classification benchmarks and five backbone models show that the proposed methods reduce the error rate over Mixup variants by 31.3%, especially in low-resource conditions. |
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) improves capabilities of dialogue systems . but they lack communication skills, which make them more like information seeking tools . |
| Approach: | They propose to empower LLMs with communication skills through inner monologues . they use a benchmark to evaluate the dialogue generation ability of the model . |
| Outcome: | The proposed model outperforms the baselines in the evaluation of communication skills. |
Copied to clipboard
| Challenge: | Proprietary and closed APIs are impacting the practical applications of natural language processing. |
| Approach: | They propose a scenario where a pre-trained model is served through a gated API . they propose 'transductive inference' that leverages statistics of unlabelled data . |
| Outcome: | The proposed model performs a few-shot classification task with unlabelled data using a gated API . the proposed model can be used to perform the task with a handful of classes . |
Copied to clipboard
| Challenge: | Prior research on training grounded factuality classification models to detect hallucinations in large language models (LLMs) has relied on public natural language inference (NLI) data and synthetic data. |
| Approach: | They propose a method that leverages multi-hop reasoning on context graphs extracted from documents to generate complex multi-level claims without relying on LLMs to decide data labels. |
| Outcome: | The proposed model outperforms GPT-4-o on the LLM-Aggrefact benchmark with much smaller model size. |
Copied to clipboard
| Challenge: | Long trajectories in deep research often exceed model context limits, compressing token budgets for both evidence collection and report writing. |
| Approach: | They propose a file-system-based framework that scales deep research beyond context window . a Context Builder agent acts as a librarian and a Report Writer agent composes the final report . |
| Outcome: | Experiments on two open-ended benchmarks show that FS-Researcher achieves state-of-the-art report quality across different backbone models. |
Copied to clipboard
| Challenge: | Recent studies have demonstrated that inference-time scaling increases performance of Large Language Models (LLMs) in various reasoning tasks such as mathematics and complex question answering by increasing the length of Chain-of-Thought (CoT). |
| Approach: | They propose a model which synthesizes longer CoT data and iteratively improves performance through self-training by incorporating a few demonstration examples. |
| Outcome: | The proposed model achieves an average improvement of more than +2.5 points across five reasoning tasks: MMLU, GSM8K, ARC-C, HellaSwag, and BBH on two backbone models. |
Copied to clipboard
| Challenge: | Existing methods for nutrition question answering face limited reasoning capacity and contextual overload . poor dietary patterns are associated with more than 11 million deaths in 2017 . |
| Approach: | They propose a framework that enables supervised multi-agent collaboration for nutritional QA. |
| Outcome: | The proposed framework outperforms single-agent and ensemble baselines in multi-agency reasoning tasks. |
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) methods are efficient for a large language model with reduced computational costs. |
| Approach: | They propose a layer-wise expert numbers and ranks allocation strategy with GuidedSelection Vectors. |
| Outcome: | The proposed method achieves superior or comparable performance to all baselines on three backbone models. |
Copied to clipboard
| Challenge: | Existing studies have focused on zero-shot cross-lingual transfer . mBERT, mBART and mT5 provide high-quality representations for texts in various languages . |
| Approach: | They propose to use mBART and NLLB-200 to finetune a multilingual pretrained language model on input-output pairs in one language and use it to make task predictions for inputs in other languages. |
| Outcome: | The proposed approach significantly reduces generation in the wrong language with full finetuning and can be competitive in some cases. |
Copied to clipboard
| Challenge: | Existing studies decompose complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. |
| Approach: | They propose to use code comments as natural logic pivot between natural language and code language to boost the code generation ability of code LLMs. |
| Outcome: | The proposed method significantly improves the code pass rate on humanEval and MBPP, while the robustness of the logical comment decoding strategy is higher than the Chain-of-thoughts prompting. |
Copied to clipboard
| Challenge: | Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task. |
| Approach: | They propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures. |
| Outcome: | The proposed method improves the identification of high-level structures using phrase-guided masking. |
Copied to clipboard
| Challenge: | Existing text-to-image models struggle to generate images with legible visual texts . current models lack support for Chinese texts, misspelling, and lack of diversity . |
| Approach: | They propose to empower backbone models to generate visual texts in Chinese and English . they propose to augment conventional training objective with glyph-aware training losses . |
| Outcome: | The proposed methods can generate visual texts in English and Chinese while maintaining image generation quality. |
Copied to clipboard
| Challenge: | Existing methods for alleviating hallucinations require costly human annotations . Existing approaches focus on a specific type of hallucinism, which limits their effectiveness . |
| Approach: | They propose a method to detect hallucinations from errors in semantic frame, discourse and content verifiability in LLM summarization using HAllucination Diversity-Aware Sampling. |
| Outcome: | The proposed framework reduces the need for costly human annotations to correct hallucinations in LLM outputs. |
Copied to clipboard
| Challenge: | Existing studies show that some parameters in pre-trained language models can be pruned away without severe accuracy degradation. |
| Approach: | They propose a method which generates more features with very cheap operations from the remaining features and can be applied to unpruned BERT models to enhance their performance. |
| Outcome: | Empirical results on the GLUE benchmark on three backbone models (i.e., BERT, RoBERTa and ELECTRA) verify the efficacy of the proposed method. |
Copied to clipboard
| Challenge: | generative models struggle to distinguish subtle differences among retrieved knowledge records, resulting in suboptimal quality of generated responses. |
| Approach: | They propose to use maximum marginal likelihood to train a perceptive retriever by utilizing signals from response generation for supervision. |
| Outcome: | The proposed approach improves on three task-oriented dialogue datasets using T5 and ChatGPT as the backbone models. |
Copied to clipboard
| Challenge: | Existing work on large-scale corpora-based language models is limited and hard to generalize to all types of pre-trained language models. |
| Approach: | They propose a two-stage SimOAP strategy that over-samples and post-evaluates large-scale responses from existing models and selects a good response based on multiple evaluation metrics. |
| Outcome: | The proposed strategy outperforms baseline and automatic evaluation strategies in both automatic and human evaluations. |
Copied to clipboard
| Challenge: | Existing research expands the tool arrays of large language models (LLMs), but the necessity of using these tools is often overlooked, leading to indiscriminate tool invocation. |
| Approach: | They propose a meta-cognition proxy proxy for LLMs self-assessment of their capabilities, reflecting the model’s awareness of its own limitations. |
| Outcome: | The proposed strategy is fine-tuned-free and costs minimal. |
Copied to clipboard
| Challenge: | Existing textual backdoor attacks focus on generating stealthy triggers or modifying model weights. |
| Approach: | They propose a Trojan Attention Loss (TAL) which enhances the Trojan behavior by directly manipulating attention patterns. |
| Outcome: | The proposed method improves the effectiveness of the backdoor attacks on different backbone models and tasks. |
Copied to clipboard
| Challenge: | Existing methods to train LLMs suffer from overthinking, leading to lengthy reasoning traces . Existing approaches to train large language models suffer from this problem . |
| Approach: | They propose a method to combine multiple reasoning chains for training LLMs . they use stepwise exploration and long-short switched sampling to evaluate reasoning paths . |
| Outcome: | The proposed method reduces reasoning lengths by approximately 30-50% . it also maintains or improves reasoning accuracy compared to baselines . |
Copied to clipboard
| Challenge: | Existing methods based on pre-trained embeddings can be easily biased by surface features that are not important for the target task. |
| Approach: | They propose a skill-based few-shot selection method for in-context learning . it generates skill-specific descriptions for each test case and candidate example . |
| Outcome: | The proposed method significantly outperforms existing methods in five cross-domain semantic parsing datasets and six backbone models. |
Copied to clipboard
| Challenge: | Existing methods to use table pre-training to boost tabular prediction performance remain open . a bachelor's degree earns less than 50K, and a generative LM can be used to unify tasks via one LM. |
| Approach: | They propose a method that leverages table pre-training to empower tabular prediction models. |
| Outcome: | The proposed method outperforms baseline models on 12 datasets and can be easily combined with various backbone models. |
Copied to clipboard
| Challenge: | Existing Large Reasoning Models (LRMs) lack explainability and controllability . Existing models target isolated levels without unification, while relying on RL . |
| Approach: | They propose an explainable, controllable, and unified reasoning framework driven by MoN. |
| Outcome: | The proposed framework achieves performance gains of 27.0% while reducing token consumption by 19.6% 63.3%. |
Copied to clipboard
| Challenge: | Existing approaches for improving LLM reasoning remain episodic and lack reusable meta-reasoning skills. |
| Approach: | They propose a framework that consolidates metacognitive experience from past reasoning episodes into reusable knowledge that improves future meta-reasoning. |
| Outcome: | The proposed framework consolidates metacognitive experience from past reasoning episodes into reusable knowledge that improves future meta-reasoning. |
Copied to clipboard
| Challenge: | Visual programming languages (VPLs) allow users to create programs through graphical interfaces, which results in easier accessibility and widespread usage in various domains. |
| Approach: | They propose to train VPLs from user instructions using large language models . they propose to use retrieval-augmented fine-tuning to leverage repetitive use of subroutines . |
| Outcome: | The proposed method outperforms prompting-based methods for LD generation accuracy even with smaller backbone models. |
Copied to clipboard
| Challenge: | Existing methods for storing and retrieving memory are limited by shallow semantic retrieval. |
| Approach: | They propose a memory mechanism that organizes and retrieves past experiences to support decision-making. |
| Outcome: | Experiments on LoCoMo and NarrativeQA show that CompassMem improves retrieval and reasoning performance across multiple backbone models. |
Copied to clipboard
| Challenge: | Recent Large Reasoning Models (LRMs) have demonstrated remarkable success in complex reasoning tasks. |
| Approach: | They propose a self-guided efficient reasoning framework that reduces FoE by pruning subs. |
| Outcome: | The proposed model outperforms eight competitive baselines while reducing token consumption by 37.7% 70.4%. |
Copied to clipboard
| Challenge: | Existing benchmarks and evaluation protocols focus on surface-level factual recall. |
| Approach: | They propose a benchmark for assessing cognitive memory under cue–trigger semantic disconnect. |
| Outcome: | The proposed framework reveals failures not captured by existing benchmarks. |
Copied to clipboard
| Challenge: | Existing models struggle to address two major bottlenecks in Document-level Relation Extraction: extreme class imbalance and complexity of multi-hop reasoning. |
| Approach: | They propose a method that decouples the extraction space into dense and sparse scenarios. |
| Outcome: | The proposed approach yields consistent improvements over various backbone models and achieves advanced performance compared to existing enhancement methods. |
Copied to clipboard
| Challenge: | Existing studies on textual plan generation only focus on LLMs, enabling applications in robotics, virtual assistants, and instruc. |
| Approach: | They propose a framework that generates and refines text-image plans step-by-step . they collect a new benchmark consisting of 1,100 tasks and their text- image pair solutions covering 11 daily topics. |
| Outcome: | The proposed framework generates and refines text-image plans step-by-step and improves on existing models. |
Copied to clipboard
| Challenge: | Large language models suffer from knowledge gaps and hallucinations, resulting in incorrect or poor reasoning. |
| Approach: | They propose Graph retrieval-augmented generation (GraphRAG) which integrates structured knowledge from external graphs to enhance model's reasoning. |
| Outcome: | Experiments on knowledge graph QA tasks show that GraphRAG significantly improves reasoning performance across multiple backbone models. |
Copied to clipboard
| Challenge: | Existing approaches focus on merging language models tuned on single objectives . existing approaches ignore the impacts of competing objectives on model tuning . |
| Approach: | They propose a model merging approach that seeks a series of backbone models and merges them according to user preferences. |
| Outcome: | The proposed approach exhibits strong controllability and Pareto optimality in controllable multi-objective generation. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved remarkable performance across NLP tasks . however, in long-context scenarios, they face high computational cost and information redundancy. |
| Approach: | They propose an encoder-decoder context compression framework that generates a compact sequence of soft tokens for downstream tasks. |
| Outcome: | Experiments show that GMSA outperforms baselines on multiple long-context question answering and summarization benchmarks while maintaining low end-to-end latency. |
Copied to clipboard
| Challenge: | Multi-domain machine translation (MDMT) is a unique challenge due to varying levels of linguistic complexity across domains. |
| Approach: | They propose a resource-rational framework that learns to modulate inference between intuitive and deliberate reasoning. |
| Outcome: | Evaluated on 15 benchmarks spanning in-domain and out-of-domain settings, as well as 3 seen and 59 unseen languages, with ablations across three backbone models, TwT-7B and Twt-14B outperform much larger SOTA reasoning models in translation quality, while reducing token usage by 32–60%. |
Copied to clipboard
| Challenge: | Mobile GUI agents show promise in automating tasks but face significant generalization challenges in long-tail scenarios. |
| Approach: | They propose a benchmark framework for mobile GUI agents that measures the performance of GUI agents by analyzing their performance. |
| Outcome: | The LearnGUI benchmark outperforms existing methods in offline and online evaluations and demonstrates consistent gains across model architectures. |
Copied to clipboard
| Challenge: | Existing approaches to compress CoT with external compressors fail to align with the model’s internal reasoning dynamics, resulting in the loss of critical logical steps. |
| Approach: | They propose a framework that exploits the model’s intrinsic saliency to compress CoT by exploiting its reasoning termination token . |
| Outcome: | The proposed framework reduces redundancy in reasoning chain by exploiting the model’s intrinsic saliency. |